Vaccines: Many Beliefs Study 3: Analysis

Data prep

Score distributions

First, let’s look at the distributions of scores for all scales, pre- and post-test:

It looks like there’s a lot of overlap in the pre- and post-test distrubtions for all scales - though a few hints of subtle shifts (e.g., in diseaseRare, vaccIntent).

Visual comparison of pre- vs. post-intervention

Let’s compare participants’ responses pre- vs. post-intervention on all scales. I’ll plot both mean pre- and post-intervention scores, as well as mean difference scores (note separate axes):

Scale for 'y' is already present. Adding another scale for 'y', which will
replace the existing scale.
Scale for 'colour' is already present. Adding another scale for 'colour',
which will replace the existing scale.

Some interesting things might be going on here! Probably most useful paired with some actual stats…

Regression analyses

First, let’s choose how to code our data - I’ve set up contrast coding, effect coding and dummy coding options here, and I’ll go with dummy-coding for now (with noInterv and pre as the baselines for comparison for condition and phase, respectively).

            DR_none
noInterv          0
diseaseRisk       1
     post_pre
pre         0
post        1

By scale

This is rather crazy (and of course exploratory) endeavor, and very vulnerable to multiple comparisons… but let’s look at the effects on all scales individually.

Beliefs about vaccines

Intentions to vaccinate (vaccIntent)

NOTE: This is our main DV of interest.

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "vaccIntent")

REML criterion at convergence: 2556

Scaled residuals: 
     Min       1Q   Median       3Q      Max 
-2.85092 -0.39910  0.07441  0.35054  2.56529 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 1.3847   1.1768  
 Residual             0.2012   0.4485  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                               Estimate Std. Error t value
(Intercept)                     1.56052    0.08250  18.915
phasepost_pre                  -0.02918    0.04155  -0.702
conditionDR_none               -0.02975    0.11360  -0.262
phasepost_pre:conditionDR_none  0.15303    0.05722   2.674

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.252              
condtnDR_nn -0.726  0.183       
phspst_:DR_  0.183 -0.726 -0.252

Success! In the form of a significant interaction between phase and condition (phasepost_pre:conditionDR_none: change from pre- to post-intervention in the Disease Risk vs. No Intervention conditions).

An analysis with change scores (a la Horne, Powell, et al. (2015, PNAS)):

Call:
lm(formula = diff ~ condition, data = d_scored %>% filter(scale == 
    "vaccIntent") %>% spread(phase, mean) %>% mutate(diff = post - 
    pre))

Residuals:
    Min      1Q  Median      3Q     Max 
-2.3239 -0.3239 -0.1239  0.2292  2.4292 

Coefficients:
                 Estimate Std. Error t value Pr(>|t|)   
(Intercept)      -0.02918    0.04155  -0.702  0.48280   
conditionDR_none  0.15303    0.05722   2.674  0.00773 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.6343 on 491 degrees of freedom
Multiple R-squared:  0.01436,   Adjusted R-squared:  0.01235 
F-statistic: 7.153 on 1 and 491 DF,  p-value: 0.007734

Again, success!

Here’s the relevant plot from the PNAS paper:

Vaccine danger (vaccDanger)

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "vaccDanger")

REML criterion at convergence: 2093.5

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.2083 -0.4591  0.0028  0.4205  3.4583 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 1.0471   1.0233  
 Residual             0.1059   0.3254  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                               Estimate Std. Error t value
(Intercept)                    -1.03519    0.07035 -14.716
phasepost_pre                  -0.01631    0.03015  -0.541
conditionDR_none                0.07442    0.09687   0.768
phasepost_pre:conditionDR_none -0.15292    0.04151  -3.684

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.214              
condtnDR_nn -0.726  0.156       
phspst_:DR_  0.156 -0.726 -0.214

Vaccine efficacy (vaccEff)

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "vaccEff")

REML criterion at convergence: 2116.6

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.8279 -0.4203  0.0566  0.4337  3.0855 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 0.7794   0.8828  
 Residual             0.1435   0.3788  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                               Estimate Std. Error t value
(Intercept)                     0.42060    0.06293   6.683
phasepost_pre                   0.05408    0.03509   1.541
conditionDR_none               -0.04291    0.08666  -0.495
phasepost_pre:conditionDR_none  0.12746    0.04832   2.638

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.279              
condtnDR_nn -0.726  0.202       
phspst_:DR_  0.202 -0.726 -0.279

Vaccines’ tendency to strain the infant immune system (vaccStrain)

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "vaccStrain")

REML criterion at convergence: 2682

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.8991 -0.4107 -0.0029  0.4120  3.4686 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 1.2621   1.1234  
 Residual             0.2759   0.5252  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                               Estimate Std. Error t value
(Intercept)                    -0.57167    0.08124  -7.036
phasepost_pre                   0.07811    0.04866   1.605
conditionDR_none                0.05090    0.11187   0.455
phasepost_pre:conditionDR_none -0.17657    0.06701  -2.635

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.299              
condtnDR_nn -0.726  0.217       
phspst_:DR_  0.217 -0.726 -0.299

Vaccine toxicity (vaccTox)

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "vaccTox")

REML criterion at convergence: 2386.2

Scaled residuals: 
     Min       1Q   Median       3Q      Max 
-2.91670 -0.40646 -0.03804  0.40448  2.94732 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 1.250    1.1180  
 Residual             0.159    0.3988  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                                Estimate Std. Error t value
(Intercept)                    -0.228326   0.077765  -2.936
phasepost_pre                  -0.048927   0.036948  -1.324
conditionDR_none                0.014480   0.107083   0.135
phasepost_pre:conditionDR_none -0.008765   0.050878  -0.172

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.238              
condtnDR_nn -0.726  0.173       
phspst_:DR_  0.173 -0.726 -0.238

Beliefs about diseases

Disease severity (diseaseSevere)

NOTE: This is the scale that we thought the Disease Risk intervention should affect most directly.

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "diseaseSevere")

REML criterion at convergence: 2398.5

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.7530 -0.3823  0.0517  0.4004  2.6982 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 0.7094   0.8423  
 Residual             0.2585   0.5084  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                               Estimate Std. Error t value
(Intercept)                     1.84721    0.06445  28.659
phasepost_pre                  -0.00515    0.04711  -0.109
conditionDR_none               -0.06721    0.08875  -0.757
phasepost_pre:conditionDR_none  0.32515    0.06486   5.013

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.365              
condtnDR_nn -0.726  0.265       
phspst_:DR_  0.265 -0.726 -0.365

Disease rarity (diseaseRare)

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "diseaseRare")

REML criterion at convergence: 2603.9

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.9595 -0.4748 -0.0016  0.4434  3.6064 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 0.7478   0.8648  
 Residual             0.3558   0.5965  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                               Estimate Std. Error t value
(Intercept)                    -0.96223    0.06882 -13.981
phasepost_pre                  -0.08326    0.05527  -1.507
conditionDR_none               -0.18238    0.09477  -1.924
phasepost_pre:conditionDR_none -0.17828    0.07610  -2.343

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.402              
condtnDR_nn -0.726  0.292       
phspst_:DR_  0.292 -0.726 -0.402

Beliefs about infants’ immune systems

Limited capacity of infants’ immune systems (infantImmLimCap)

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "infantImmLimCap")

REML criterion at convergence: 2759.8

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.5529 -0.3988 -0.0115  0.4562  3.7260 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 0.9137   0.9559  
 Residual             0.4054   0.6367  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                               Estimate Std. Error t value
(Intercept)                     0.41373    0.07524   5.499
phasepost_pre                  -0.03433    0.05899  -0.582
conditionDR_none                0.07857    0.10361   0.758
phasepost_pre:conditionDR_none  0.11433    0.08123   1.408

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.392              
condtnDR_nn -0.726  0.285       
phspst_:DR_  0.285 -0.726 -0.392

Weakness of infants’ immune systems (infantImmWeak)

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "infantImmWeak")

REML criterion at convergence: 2785.1

Scaled residuals: 
     Min       1Q   Median       3Q      Max 
-2.56604 -0.47750 -0.00131  0.45358  2.41157 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 1.0021   1.001   
 Residual             0.3969   0.630   
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                               Estimate Std. Error t value
(Intercept)                     0.29013    0.07749   3.744
phasepost_pre                  -0.04549    0.05837  -0.779
conditionDR_none                0.10449    0.10670   0.979
phasepost_pre:conditionDR_none  0.04242    0.08037   0.528

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.377              
condtnDR_nn -0.726  0.274       
phspst_:DR_  0.274 -0.726 -0.377

Other beliefs, attitudes, and worldviews

Medical skepticism (medSkept)

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "medSkept")

REML criterion at convergence: 2453.1

Scaled residuals: 
     Min       1Q   Median       3Q      Max 
-3.10928 -0.43517 -0.02145  0.42521  3.09386 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 1.1431   1.0692  
 Residual             0.1953   0.4419  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                               Estimate Std. Error t value
(Intercept)                     0.05007    0.07579   0.661
phasepost_pre                  -0.04220    0.04094  -1.031
conditionDR_none                0.05634    0.10436   0.540
phasepost_pre:conditionDR_none -0.11677    0.05638  -2.071

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.270              
condtnDR_nn -0.726  0.196       
phspst_:DR_  0.196 -0.726 -0.270

Holistic balance (hb)

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "hb")

REML criterion at convergence: 2407.1

Scaled residuals: 
     Min       1Q   Median       3Q      Max 
-2.53977 -0.42672  0.03449  0.46098  2.53965 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 1.0026   1.0013  
 Residual             0.2001   0.4473  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                               Estimate Std. Error t value
(Intercept)                     0.22232    0.07185   3.094
phasepost_pre                  -0.01545    0.04144  -0.373
conditionDR_none                0.00999    0.09893   0.101
phasepost_pre:conditionDR_none -0.11840    0.05707  -2.075

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.288              
condtnDR_nn -0.726  0.209       
phspst_:DR_  0.209 -0.726 -0.288

Naturalism (nat)

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "nat")

REML criterion at convergence: 2450.2

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.0499 -0.4451  0.0037  0.4351  3.3838 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 1.0731   1.0359  
 Residual             0.2049   0.4526  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                               Estimate Std. Error t value
(Intercept)                    -0.34335    0.07406  -4.636
phasepost_pre                  -0.03004    0.04194  -0.716
conditionDR_none                0.02027    0.10198   0.199
phasepost_pre:conditionDR_none -0.05778    0.05775  -1.001

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.283              
condtnDR_nn -0.726  0.206       
phspst_:DR_  0.206 -0.726 -0.283

Overparenting (overpar)

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "overpar")

REML criterion at convergence: 2593.6

Scaled residuals: 
     Min       1Q   Median       3Q      Max 
-3.15393 -0.47665 -0.06009  0.45377  2.49500 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 0.7761   0.8810  
 Residual             0.3408   0.5837  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                                Estimate Std. Error t value
(Intercept)                    -0.015451   0.069235  -0.223
phasepost_pre                  -0.019742   0.054083  -0.365
conditionDR_none               -0.001472   0.095338  -0.015
phasepost_pre:conditionDR_none  0.284358   0.074473   3.818

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.391              
condtnDR_nn -0.726  0.284       
phspst_:DR_  0.284 -0.726 -0.391

Parental Expertise (parentExpert)

Linear mixed model fit by REML ['lmerMod']
Formula: mean ~ phase * condition + (1 | workerId)
   Data: d_scored %>% filter(scale == "parentExpert")

REML criterion at convergence: 2446

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.2300 -0.4653 -0.0184  0.4962  3.0384 

Random effects:
 Groups   Name        Variance Std.Dev.
 workerId (Intercept) 1.2031   1.0969  
 Residual             0.1844   0.4294  
Number of obs: 986, groups:  workerId, 493

Fixed effects:
                               Estimate Std. Error t value
(Intercept)                    -0.49356    0.07717  -6.396
phasepost_pre                   0.01459    0.03978   0.367
conditionDR_none                0.01741    0.10626   0.164
phasepost_pre:conditionDR_none -0.10613    0.05478  -1.937

Correlation of Fixed Effects:
            (Intr) phsps_ cndDR_
phasepst_pr -0.258              
condtnDR_nn -0.726  0.187       
phspst_:DR_  0.187 -0.726 -0.258

Other stuff

Violin plots

With violin plots, we can see the full distribution of scores at each time point in each condition:

Derek’s analyses

DATE: March 13, 2018 3:52 PM

I’m going to pick up here and do some further regression analyses. I’ll look at this data in the way I wish we’d looked at the original PNAS data.

That is, (1) using an ordinal HLM regression over the five separate scale items (conceptually, I think, similar to SEM-style approaches), and (2) using beta regression.

There are two general model forms that I think are reasonable for looking at this. First:

response ~ phase * condition

Kara already tried this general approach, saving the model as r1_vaccIntent. So the only difference here will be the distributions I use. In this case, the interaction term is required, and the real thing to test is the phase*condition interaction.

And second:

post_response ~ pre_response * condition

Here pre_response could be the response on the overall pre-test scale, or the individual items. In addition, the interaction terms are optional, as warranted by the data. I’d wager this is the more familiar approach for most psychologists.

Ordinal HLM

Predicting “response”

Predicting post-test

Here pre is the specific item pre-test score and preMean is the mean of the scale at pretest.

Regression on scale averages

Recall, Kara already looked at a normal regression on responses with r1_vaccIntent model.

Normal regression: predicting post-test

            diseaseRisk
noInterv              0
diseaseRisk           1

Beta regression: response

A beta regression on the responses accords with the linear regression on responses–slight positive effect of diseaseRisk, slight negative effect of autism correction, but too much uncertainty overall.

But, our response variable isn’t really all that appropriate for a linear regression, even if many researchers would be happy with that results and move on. Instead, vaccIntent is bounded and highly skewed. That makes it suitable for beta regression.

Error in as.vector(res) : object 'res' not found

Beta regresstion: posttest

Let’s look at predicting post-test with beta regression. Here there’s no evidence at all for any positive effect of diseaseRisk (in fact the coefficient is negative), but there is evidence of BACKFIRE for the autism correction condition.

Call:
betareg(formula = post ~ scale(pre) * condition, data = d2 %>% mutate(post = rescale_beta(post, 
    -3, 3)))

Standardized weighted residuals 2:
    Min      1Q  Median      3Q     Max 
-7.4139 -0.6774 -0.2381  0.6395  4.6225 

Coefficients (mean model with logit link):
                                Estimate Std. Error z value Pr(>|z|)    
(Intercept)                      1.38527    0.05396  25.674   <2e-16 ***
scale(pre)                       1.17727    0.05586  21.073   <2e-16 ***
conditiondiseaseRisk             0.15757    0.07167   2.198   0.0279 *  
scale(pre):conditiondiseaseRisk -0.06144    0.07196  -0.854   0.3932    

Phi coefficients (precision model with identity link):
      Estimate Std. Error z value Pr(>|z|)    
(phi)   8.6048     0.5632   15.28   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

Type of estimator: ML (maximum likelihood)
Log-likelihood: 583.5 on 5 Df
Pseudo R-squared: 0.5531
Number of iterations: 12 (BFGS) + 3 (Fisher scoring) 

AIC favors the beta regression over the linear regression strongly.

[1] -864.451
[1] -1157.006